Model Invocation for Three Dimensional Scene Understanding

Any gantral model-based vision system must somehow se­ lect a few serious candidates from its model base before ap­ plying model-directed processing. This is necessary for both efficiency and recognising 'similar* models (i.e. handling data errors, generic models and previously unseen objects). This pa­ per shows how one can Integrate knowledge of object properties, structural and generic relations to create a network computa­ t ion that performs model invocation. The paper demonstrates successful invocation in a scene containing a self and externally obscured P U M A robot. 1 In t r o d net ion One important and difficult task for a general model based vision sys­ tem is invoking the correct model. Because of the potentially huge number of possible objects, it is imperative that only a few serious candidates are selected for detailed consideration. Visual understand­ ing must also include a pre-attentive element, because all models need be considered, yet active, direct comparison is computationally infeasible. Further, previously unseen objects, flexible objects seen in new configurations, incompletely visible objects (e.g. occlusion) and object variants (e.g. flaws, generics, new exemplars, etc.) require selecting models that are "close" to the data. Model invocation is not just a visual problem (e.g. integrating cues while doing crossword pussies or invoking situation schemas), but here only the visual problem is considered. Invocation associates clues that suggest rather than verify. It may support the "seeing* of nonexistent, but highly plausible objects, as in surrealist art. To date, little work has been done on sophisticated model invo­ cation in the context of 3D vision. Using easily measured properties to select potential models fails in large model bases, because many objects share similar properties. Further, data errors, generic objects, object substructure and occlusion complicate indexing. Arbib [l] proposed a schema-based invocation process with acti­ vation levels based on evidence, competition and cooperation from related activity. Marr [6) considered direct search in a model-base linked using specificity, adjunct and parent relations. Hinton and Lang [5] evaluated a connectionist model of invocation for 2D models, treating both model and data feature evidence identically. Feldman and Ballard [2] proposed a detailed computational model integrating evidence from spatially coincident property pairings. This paper describes a solution that builds on these, embodying ideas on parallel networks, object description and representation in the context of 3D models and 3D visual information. The result is a plausibility calculation in a network structured according constraints defined by the structural, generic and context relationships. Acknowledgements Portions of this work were supported by a University of Edinburgh studentship. Thanks go to R. Beattie, J. Hallam, D. Hogg, J. Howe, M. Orr and many others. 2 Problem Context These results are from the IMAGINE project [3] which investigated recognising 3D objects starting from 3D scene information. Earlier stages of processing include: 1. exploiting 3D feature continuity to overcome occlusion, 2. grouping individual surfaces to form primitive and depth aggre­ gated surface clusters [4]. 3. describing the significant scene features (curves, surfaces and volumes) by their 3D properties. Model invocation happens at this point. After invocation, modeldirected processes orient and verify the hypotheses. Recognition starts from 2 1/2D sketch -like data segmented into sur­ face patches of nearly uniform shape and separated by various shape or obscuring boundaries. As no well-developed processes produce this data yet, the program input is from computer augmented, handsegmented test images. The paper illustrates the invocation process using a test image of a PUMA robot with its gripper obscured, using the surfaces shown in figure 1. The scene has flexibly connected rigid solids, has a variety of curved surfaces and the robot is both externally and self-obscured. Object models are primarily structural, with features attached by reference frame transformations. The main primitives are the sur­ face patch (SURFACE), characterised by consistent surface shape and polycurve boundary, and hierarchical groupings of surfaces (ASSEM­ BLY). Four additional representations are added for invocation: 1. generic relationships between models, 2. major features of each assembly grouped according to viewpoint, 3. relevant properties with typical values for each structure, and 4. weighting factors modifying the importances of the properties and relationships. Figure 1: Test Scene Surface Regions